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ON  THE  LOCAL  CONVERGENCE  OF  PATTERN  SEARCH* * * § 

ELIZABETH  D.  DOLANt,  ROBERT  MICHAEL  LEWIS*,  AND  VIRGINIA  TORCZON§ 


Abstract.  We  examine  the  local  convergence  properties  of  pattern  search  methods,  complementing  the 
previously  established  global  convergence  properties  for  this  class  of  algorithms.  We  show  that  the  step-length 
control  parameter  which  appears  in  the  definition  of  pattern  search  algorithms  provides  a  reliable  asymptotic 
measure  of  first-order  stationarity.  This  gives  an  analytical  justification  for  a  traditional  stopping  criterion 
for  pattern  search  methods.  Using  this  measure  of  first-order  stationarity,  we  analyze  the  behavior  of  pattern 
search  in  the  neighborhood  of  an  isolated  local  minimizer.  We  show  that  a  recognizable  subsequence  converges 
r-linearly  to  the  minimizer. 

Key  words,  pattern  search,  local  convergence  analysis,  global  convergence  analysis,  desultory  rate  of 
convergence 

Subject  classification.  Applied  and  numerical  mathematics 

1.  Introduction.  Pattern  search  methods  are  a  class  of  direct  search  methods  for  solving  nonlinear 
optimization  problems.  The  global  convergence  properties  of  pattern  search  for  both  constrained  and  un¬ 
constrained  problems  have  been  established  in  a  series  of  papers  [7—11].  In  this  paper,  we  consider  the  local 
convergence  properties  of  pattern  search  and  revisit  the  global  convergence  properties  in  light  of  these  new 
results. 

For  simplicity,  our  discussion  will  focus  on  the  case  of  unconstrained  minimization: 

min  /(x). 
xenn 

Results  similar  to  those  we  present  here  can  also  be  derived  for  the  general  case  of  bound  and  linear  constraints 
[9, 10].  However,  the  underlying  ideas  are  simpler  to  explain  for  the  unconstrained  case. 

We  first  show  how  the  pattern  size  parameter,  which  plays  a  central  role  in  the  definition  of  pattern 
search  methods  and  tacitly  serves  as  a  step-length  control  mechanism,  also  provides  a  reliable  asymptotic 
measure  of  first-order  stationarity.  This  gives  an  analytical  justification  for  the  traditional  use  of  the  pattern 
size  parameter  as  a  stopping  criterion.  We  also  establish  a  local  convergence  result  concerning  the  behavior 
of  the  sequence  of  iterates  produced  by  a  pattern  search  algorithm  in  the  neighborhood  of  an  isolated  local 
minimizer  x*.  We  show  that  under  reasonable  hypotheses  the  sequence  of  iterates  converges  to  x*  and, 
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moreover,  an  identifiable  subsequence  of  the  iterates  converges  r-linearly  to  x* .  These  analytical  results  are 
illustrated  with  some  simple  numerical  experiments  on  quadratic  objectives. 

What  is  interesting  about  the  analysis  presented  here  is  that  we  can  establish  local  convergence  properties 
despite  the  fact  that  direct  search  methods  do  not  employ  an  explicit  representation  of  the  gradient  of  the 
objective  and,  as  a  consequence,  cannot  enforce  a  notion  of  sufficient  decrease.  We  proved  global  convergence 
results  for  pattern  search  by  showing  that  all  iterates  lie  on  a  rational  lattice.  It  is  this  restriction  on  the  form 
of  the  steps  that  allows  us  to  relax  the  notion  of  sufficient  decrease  and  yet  still  prove  global  convergence. 
Pattern  search  may  accept  any  point  on  the  current  lattice  so  long  as  it  produces  simple  decrease  on  the  value 
of  the  function  at  the  current  iterate.  However,  key  to  the  global  analysis  is  the  notion  of  having  searched 
in  a  sufficient  number  of  directions  from  the  current  iterate  to  guarantee  that  we  have  not  overlooked  a 
potential  direction  of  descent.  It  is  only  after  searching  over  a  sufficient  set  of  directions  that  we  are  allowed 
to  reduce  the  current  step-length  control  parameter — which  has  the  effect  of  refining  the  lattice  over  which 
we  are  searching. 

This  notion  of  sufficient  local  information  at  iterations  at  which  we  reduce  the  pattern  size  allows  us  to 
show  that  the  pattern  size,  as  measured  by  the  step-length  control  parameter,  provides  a  reliable  asymptotic 
measure  of  first-order  stationarity.  This  analytical  result  is  gratifying  since  it  vindicates  the  long-standing 
use  of  the  step-length  control  parameter  as  a  stopping  criterion  for  direct  search  methods  (see,  for  instance, 
Section  4  of  [6]).  The  result  on  the  correlation  of  the  pattern  size  parameter  and  stationarity  then  enables 
us  to  study  the  local  convergence  properties. 

Notation.  We  use  L{x o)  to  denote  the  set  {x  \  f(x)  <  f(x o)}.  It  is  assumed,  unless  otherwise  noted, 
that  all  norms  are  Euclidean  vector  norms  or  the  associated  operator  norm.  We  will  use  d  to  denote  the 
boundary  of  a  given  set.  Given  x  and  r  >  0,  we  denote  by  B(x1r)  the  open  ball  of  radius  r  centered  at  x: 
B{x,r)  =  {y  |  ||  y-x  ||  <r  }. 

2.  Pattern  search.  We  first  review  the  elements  of  pattern  search  that  play  a  role  in  our  local  analysis. 
There  are  rigorous  formal  definitions  of  pattern  search  [7, 11],  several  features  of  which  we  will  shortly  recall. 
However,  pattern  search  can  perhaps  be  most  quickly  understood  with  the  following  simple  example  of  a 
pattern  search  algorithm.  At  iteration  k ,  we  have  an  iterate  Xk  €  Rn  and  a  step-length  control  parameter 
Ak  >  0.  Let  e^,  j  =  l,...,n  be  the  standard  unit  basis  vectors.  For  the  purposes  of  this  example,  we 
represent  the  pattern  of  points  over  which  we  will  search  as  the  set  D  =  =  (ei?  •  •  • » en>  —ei, . . . ,  -en} 

though,  as  we  discuss  shortly,  many  other  choices  are  possible.  We  now  have  several  algorithmic  options 
open  to  us.  We  will  consider  the  simple  opportunistic  strategy,  which  is  to  look  successively  at  the  points 
x+  =  xk  +  A kdi,  i  €  {1, . . .  ,2 n}  until  we  either  find  an  x+  for  which  f{x+)  <  f(xk)  or  we  exhaust  all  2 n 
possibilities.  Fig.  2.1  illustrates  the  pattern  of  points  among  which  we  search  for  x+  when  n  =  2. 


Afc 


o  •  o 

Xk 


o 

Fig.  2.1.  A  simple  instance  of  pattern  search 
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If  we  find  no  x+  such  that  f(x+)  <  f(x *),  then  we  call  the  iteration  unsuccessful ;  otherwise,  we 
consider  the  iteration  successful  since  we  have  found  a  new  iterate  that  produces  decrease  on  /  at  xk> 
When  the  iteration  is  unsuccessful,  we  set  xk+i  =  Xk  and  are  required  to  reduce  Ak  (typically,  by  a  half) 
before  continuing;  otherwise,  for  a  successful  iteration,  we  set  Xk+i  =  x+  and  leave  the  step-length  control 
parameter  alone,  i.e.,  Afc+i  =  A^  (though  the  analysis  also  allows  us  to  increase  A*,  if  the  iteration  is  a 
success).  We  repeat  this  process  until  some  suitable  stopping  criterion  is  satisfied. 

Note  that  overall  our  requirements  on  the  outcome  of  the  search  at  each  iteration  are  light:  if  after 
searching  over  all  the  points  defined  by  A  fed*,  i  —  1, . .  .,2  n  we  fail  to  find  a  point  2+  =  Xk  +  A  kd>i  that 
reduces  the  value  of  /  at  xk,  then  we  must  try  again  with  a  smaller  value  of  A*.  Otherwise,  we  accept  as 
our  new  iterate  the  first  point  in  the  pattern  that  produces  decrease.  In  the  latter  case,  we  may  choose  to 
modify  Afc.  In  either  case,  we  are  free  to  make  changes  to  the  pattern  D  to  be  used  in  the  next  iteration, 
though  we  leave  the  pattern  unchanged  in  the  example  given  above.  However,  in  general,  changes  to  either 
the  step-length  control  parameter  or  the  pattern  are  subject  to  certain  algebraic  conditions,  outlined  fully 
in  [7]. 

A  distinguishing  characteristic  of  pattern  search  methods  is  that  they  sample  the  function  over  a  prede¬ 
fined  pattern  of  points,  all  of  which  lie  on  a  rational  lattice.  By  enforcing  structure  on  the  form  of  the  points 
in  the  pattern,  as  well  as  some  simple  rules  on  both  the  outcome  of  the  search  and  the  subsequent  updates, 
standard  global  convergence  results  can  be  obtained  [7, 11]. 

There  remains  the  question  of  what  constitutes  an  acceptable  pattern.  A  pattern  must  be  a  positive 
spanning  set  for  Rn  [4, 7].  A  set  of  vectors  {ai, . . .  ,ap}  positively  spans  Rn  if  any  vector  x  G  Rn  can  be 
written  as  a  nonnegative  linear  combination  of  the  vectors  in  the  set;  i.e., 

x  =  ct\a\  -I - b  CLvap  ai  >  0  Vi. 

The  set  {ai,...,ar}  is  called  positively  dependent  if  one  of  the  af s  is  a  nonnegative  combination  of  the 
others;  otherwise  the  set  is  positively  independent.  A  positive  basis  is  a  positively  independent  set  whose 
positive  span  is  Rn. 

It  is  straightforward  to  verify  that  the  set  of  vectors  {ei,e2,  —  ei,  —02}  we  used  to  define  the  pattern  for 
our  simple  example  above  is  a  positive  spanning  set,  as  is  {ei, . . . ,  e„,  -ei, . . . ,  -en}  for  other  values  of  n. 

2.1.  Prior  results.  Before  proceeding  to  our  local  convergence  results,  we  recall  the  following  propo¬ 
sition  from  [7],  which  we  will  state  here  without  proof. 

Proposition  2.1.  Given  any  set  {ai, . . .  ,ar}  that  positively  spans  Rn,  a*  /  0  for  i  =  1, . . .  ,r,  there 
exists  C2.1  >0  such  that  for  all  x  G  Rn,  we  can  find  an  a\  for  which 

xTai  >  c2.i\\x\\  H|. 

Note  that  this  is  a  purely  geometric  property  of  positive  spanning  sets. 

2.2.  Some  formal  definitions.  We  also  need  to  recall  some  notation  regarding  both  the  pattern  and 
the  form  of  the  search.  For  the  details,  we  refer  the  reader  to  [7, 11]. 

We  have  already  noted  that  the  pattern  must  be  a  positive  spanning  set  for  Rn.  In  fact,  we  represent 
the  pattern  using  two  components,  a  basis  matrix  and  a  generating  matrix. 

The  basis  matrix  can  be  any  nonsingular  matrix  B  G  RnXn. 

The  generating  matrix  is  an  integral  matrix  Ck  G  Znxpk,  where  pk  >  n  +  1.  We  require  Ck  to  contain  a 
minimum  of  n  -b  2  columns  because  the  minimum  number  of  vectors  in  a  positive  spanning  set  is  n  +  1  [4]; 
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for  convenience,  we  require  a  column  of  zeros  to  denote  the  zero  step.  We  further  partition  the  generating 
matrix  so  that  the  positive  basis  that  guarantees  that  the  pattern  positively  spans  Rn  is  revealed.  We  call  the 
columns  associated  with  the  positive  basis  the  core  pattern,  which  we  denote  as  Fk;  any  remaining  columns 
in  the  positive  spanning  set  are  denoted  using  Lk: 

(2.1)  Ck=  [  Tk  Lk  0  ]. 

We  further  require  that  Tk  €  T,  where  T  comprises  a  finite  set  of  integral  matrices,  each  of  which  is  a  positive 
basis  for  Rn. 

A  pattern  Pk  is  then  represented  by  the  columns  of  the  matrix  Pk  =  BCk.  For  convenience,  we  use  the 
partition  of  the  generating  matrix  Ck  given  in  (2.1)  to  partition  Pk  as  follows: 

Pk=  BCk  =  [  BTk  BLk  0  ]. 

To  tie  this  notation  back  to  the  example  that  introduces  Section  2,  we  note  that  B  =  I,  Tfc  =  [I  —I] 
and  Lk  =  [  0]  (where  0  denotes  the  zero  vector).  Since  the  choice  of  Tk  and  Lk  is  fixed  for  all  k,  Pk  =  [D  0] 
for  all  k. 

Now,  given  the  step-length  control  parameter  Ak  €  R,  A*,  >  0,  we  define  a  trial  step  sk  to  be  any  vector 
of  the  form  s\  =  A kBclk,  where  c{  is  a  column  of  Ck. 

In  Fig.  2.2  we  state  the  general  form  of  a  pattern  search  method  for  unconstrained  minimization. 

Let  xo  €  Rn  and  A0  >  0  be  given. 

For  k  =  0, 1, . . until  convergence  do: 

1.  Compute  f(xk). 

2.  Determine  a  step  sk  using  an  unconstrained  exploratory  moves  algorithm. 

3.  If  f(xk  +  sk)  <  f(xk),  then  xk+1  =  xk  +  sk .  Otherwise  xk+1  =  xk. 

4.  Update  Ck  and  Afc. _ 

Fig.  2.2.  Generalized  pattern  search  for  unconstrained  minimization 

We  have  remarkable  latitude  in  the  way  in  which  we  choose  the  step  sk.  For  the  global  convergence 
analysis  to  hold,  we  need  only  satisfy  the  hypotheses  on  the  outcome  of  the  unconstrained  exploratory  moves, 
given  in  Fig.  2.3. 

1.  sk  e  A kPk. 

2.  If  min{/(a:fc  +  y)  \  y  €  AkBTk}  <  f(xk ),  then  f{xk  +  sk )  <  }{xk ). _ _ 

Fig.  2.3.  Hypotheses  on  the  outcome  of  the  unconstrained  exploratory  moves 

A  few  comments  on  these  hypotheses  are  in  order.  The  first  hypothesis  is  straightforward:  the  step 
returned  must  be  defined  by  the  current  pattern  Pk ,  scaled  by  the  current  value  of  the  step-length  control 
parameter  Ak.  This  is  the  condition  that  ensures  that  the  steps  we  consider  remain  on  the  rational  lattice; 
arbitrary  steps  are  not  allowed. 

For  our  purposes,  the  second  hypothesis  is  the  more  interesting.  Notice  that  in  Fig.  2.2,  all  that  is  required 
for  a  successful  iteration  of  pattern  search  is  that  the  step  sk  produce  simple  decrease,  i.e.,  f{xk+sk)  <  f(xk ). 
Thus,  any  nonzero  step  defined  by  a  column  of  A kPk  that  satisfies  the  condition  f(xk  +  sk)  <  f{xk)  may  be 
returned  by  the  exploratory  moves  since  it  immediately  satisfies  both  of  the  hypotheses  given  in  Fig.  2.3 
even  if  we  do  not  explicitly  verify  that  min{/(a:fc  +  y)  |  y  €  A kBTk}  <  f(xk)  is  true.  Thus,  as  we  suggested 
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for  the  pattern  search  algorithm  described  in  Section  2,  as  soon  as  we  find  a  point  in  the  pattern  that  satisfies 
this  simple  decrease  criterion,  we  may  terminate  the  iteration  and  declare  it  a  success. 

Recall,  however,  that  Pk  also  contains  a  column  of  all  zeros,  so  the  question  is:  when  is  the  zero  step 
acceptable?  The  point  of  the  second  hypothesis  in  Fig.  2.3  is  to  ensure  that  we  have  sufficient  information 
about  the  local  behavior  of  /  to  declare  an  iteration  unsuccessful,  accept  the  zero  step  Sk  =  0  (so  that 
Xk+\  =  £fc),  and  reduce  to  continue  the  search  with  smaller  steps  at  the  next  iteration.  The  second 
hypothesis  implicitly  decrees  that  we  may  only  return  the  zero  step  when  we  have  looked  at  all  the  steps 
defined  by  the  core  pattern,  i.e.,  all  steps  of  the  form  y  €  A^BIV  If  none  of  the  steps  in  the  core  pattern 
produce  decrease  on  /  at  then  we  are  free  to  accept  the  zero  step.  But  if  any  step  in  the  core  pattern 
produces  descent,  the  exploratory  moves  must  return  a  step  that  produces  descent — though  as  we  have 
already  seen,  that  step  need  not  be  defined  by  A^BTk  so  long  as  it  is  defined  by  AfcBCV 

For  the  purposes  of  the  local  convergence  analysis  that  follows,  it  is  the  subsequence  of  unsuccessful 
iterates  that  interests  us.  To  accept  the  zero  step,  the  search  must  have  considered  all  the  points  defined  by 
the  core  pattern,  which  is  itself  a  positive  basis.  This  is  where  the  geometric  property  of  positive  spanning 
sets  captured  in  Proposition  2.1  comes  in.  Even  though  we  do  not  have  an  explicit  representation  of  V/(xfc) 
(assuming  that  /  is  differentiable),  Proposition  2.1  gives  us  a  positive  lower  bound,  which  is  independent  of  k , 
on  the  angle  between  V/fyV  (assuming  it  is  nonzero)  and  some  a*  in  the  positive  spanning  set — even  though 
at  any  given  iteration  we  do  not  know  for  which  az  this  lower  bound  holds.  However,  this  guaranteed  lower 
bound,  when  combined  with  the  second  hypothesis  in  Fig.  2.3,  ensures  that  at  the  end  of  an  unsuccessful 
iteration,  we  have  sufficient  information  about  the  local  behavior  of  /  at  Xk-  Furthermore,  the  quality  of  our 
local  information  improves  as  we  reduce  A*. 

Thus  we  have  enough  structure  to  construct  local  convergence  results.  The  subsequence  of  unsuccessful 
iterates  is  well-defined:  they  are  the  iterations  at  which  we  can — we  must — reduce  A k  to  ensure  that  the 
search  can  make  further  progress.  But  we  reduce  Ak  only  after  we  have  sufficient  local  information  about 
the  behavior  of  /  to  justify  this  action:  we  have  considered  all  the  steps  defined  by  the  columns  of  AkTk 
and  none  of  them  have  produced  descent  on  /  at  x/c.  This  is  the  fact  we  now  exploit. 

3.  Measuring  first-order  stationarity.  The  following  theorem  shows  that  the  step-length  control 
parameter  A*,  when  small  enough,  provides  a  reasonable  measure  of  first-order  stationarity  when  reduced 
after  an  unsuccessful  iteration.  For  simplicity,  we  assume  that  V/(x)  is  Lipschitz  continuous;  however,  for 
the  reader  interested  in  greater  generality  we  note  that  a  similar  result  can  be  proven  under  the  assumption 
of  uniform  continuity. 

Theorem  3.1.  Suppose  V/(x)  is  Lipschitz  continuous  on  an  open  neighborhood  Q  of  L(x o)  with  Lips¬ 
chitz  constant  C.  Then  there  exist  £3.1  >  0  and  C3.1  >0  for  which  the  following  holds.  If  x &  is  an  iterate  at 
which  there  is  an  unsuccessful  iteration  and  Ak  <  £3.1*  then 

||  V/(xfc)  ||  <  C3.iAfc. 


Proof  Let  r  =  |min{l,dist (0L(xo),dft)}.  If  x  €  L{x0),  then  the  ball  B{x,r)  is  contained  in  Q.  We  are 
interested  in  steps  of  the  form  s  =  A kBclk,  where  clk  is  a  column  of  the  core  matrix  IV  Since  IV  6  T  and  T 
is  finite,  ||s||  <  Afc||I?||r*,  where  T*  is  the  maximum  norm  of  any  column  of  the  matrices  in  the  set  T.  Set 
*3u  =  r/(||£||r*). 

By  the  definition  of  pattern  search,  for  any  Tfc  6  T,  the  set  {s  \  s  €  AkBTk}  forms  a  positive  basis  for 
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Rn.  Thus  Proposition  2.1  assures  us  of  the  existence  of  a  step  s  for  which 
(3.1)  -X7f(xk)Ts  >  c2.i||V/(xfc)||  ||s||. 

Since  iteration  k  is  unsuccessful,  it  follows  that 

f(xk  + s)  -  f(xk)  >  0  VseAkBTk. 

Since  Ak  <  £3.1,  ( xk  +  s)  e  B(xk ,  r)cO  and  we  can  apply  the  mean  value  theorem.  In  addition,  using  (3.1) 
and  the  Cauchy-Schwarz  inequality,  for  some  £  in  the  line  segment  connecting  Xk  and  Xjj  +  swe  have 

0  <  f(xk  +  s)  -  f(xk) 

=  Vf(xk)Ts  +  (V/«)  -  V/(a;fc))T  « 

<  -C2.i||V/(*fc)ll  INI  +  l|V/(0  -  V/(a*)ll  IMI> 

where  s  is  the  step  for  which  (3.1)  holds.  Thus 

C2.l||V/(#fc)||  <  ||V/(£)  —  V/(Xfc)||. 

Again,  since  B(xk,r)  C  ft,  the  Lipschitz  continuity  of  V/(x)  gives  us 

C3.x||V/(*fc)||  <  C||e-*fc||  <  C\\s\\  <  CAk\\B\\T*. 

Therefore 


||V/(xfc)||<c3.iAfcl 


with  c3.i  =  C\\B\\T*/c2.i-  □ 

Theorem  3.1  gives  a  theoretical  justification  for  a  traditional  stopping  criterion  for  pattern  search  meth¬ 
ods.  In  the  long  literature  on  direct  search  methods,  one  frequently  encounters  the  suggestion  that  a  direct 
search  method  be  terminated  when  some  measure  of  the  step  size  first  falls  below  a  value  deemed  suitably 
small  [2, 3, 6].  In  the  case  of  pattern  search,  Theorem  3.1  vindicates  this  intuition.  At  unsuccessful  iterations, 
the  step  size  in  pattern  search  (as  measured  by  A*.)  provides  a  bound  on  first-order  stationarity.  At  the 
same  time,  it  is  at  the  unsuccessful  iterations  that  Ak  is  decreased.  Thus,  decrease  in  Ak  provides  a  natural 
measure  of  progress  which  can  reliably  be  used  to  test  for  convergence.  We  discuss  further  the  use  of  to 
measure  progress  when  we  present  some  numerical  examples  in  Section  5. 

A  similar  relation  between  Ak  and  constrained  stationarity  in  the  case  of  pattern  search  for  bound 
constrained  problems  is  explicitly  used  in  the  pattern  search  augmented  Lagrangian  algorithm  in  [8].  The 
result  plays  a  critical  role  in  allowing  successive  inexact  minimization  of  an  augmented  Lagrangian  without 
an  explicit  estimate  of  the  gradient.  A  relation  similar  to  Theorem  3.1  for  linearly  constrained  pattern  search 
appears  in  [10]. 

The  global  convergence  analysis  of  pattern  search  in  [11]  says  that  if  L(x 0)  is  compact  then  we  have 
liminffc-00  ||  A*  ||  =  0  and  liminf^oo  ||  V/(xk)  ||  =  0.  The  former  result  and  Theorem  3.1  allow  us 
to  sharpen  the  latter  result.  Let  the  set  S  represent  a  subsequence  of  unsuccessful  iterations  for  which 
limfc_>oo ykes  Afc  =  0  (such  a  subsequence  exists  since  liminf^-^oo  A&  —  0).  Then  Theorem  3.1  says  that  we 
have  limfc_oo}fc65  II  V/(xfc)  ||  =  0. 

The  general  result  liminffc^oo  II  V/(xk)  ||  =  0  for  pattern  search  leaves  open  the  possibility  that 
II  V/(ajfc)  ||  does  not  converge.  In  [1],  Audet  shows  that  this  can  actually  occur  by  constructing  a  pat¬ 
tern  search  algorithm  and  objective  for  which  {xk}  has  infinitely  many  limit  points,  one  of  which  is  not  a 
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stationary  point  of  the  objective.  However,  Theorem  3.1  reassures  us  that  in  practice  we  need  not  worry 
about  convergence  to  non-stationary  points.  If  we  stop  the  algorithm  at  the  first  unsuccessful  iterate  for 
which  Afc  <  A*  for  some  suitably  small  stopping  tolerance  A*,  then  Theorem  3.1  says  that  ||  V/(xfc)  ||  will 
also  be  small. 

4.  Local  convergence.  We  next  apply  Theorem  3.1  to  prove  results  on  the  local  convergence  of  pattern 
search  methods.  We  place  the  following  mild  hypothesis  on  the  patterns  Pk  in  order  to  bound  the  size  of 
the  steps 

Hypothesis  0.  The  columns  of  the  pattern  matrix  Pk  —  [cj.  •  •  *c^fc]  remain  bounded  in  norm ,  i.e.,  there 
exists  C  >  0  such  that  for  all  k,  C  >  ||  c\  ||,  for  all  i  —  1,  *  •  •  Thus ,  there  exists  a  constant  c0  >  0  such 
that  any  step  Sk  satisfies 


II  Sk  ||  <  coAfc. 

We  also  impose  the  following  condition  on  the  pattern  size  parameter  A*. 

Hypothesis  1.  There  exists  N  for  which  A k  is  monotonically  nonincreasing  for  all  k>  N. 

Note  that  this  is  a  condition  we  can  explicitly  enforce  by  not  allowing  increases  in  A^  after  some  iteration 
JV;  Ak  can  stay  the  same  or  decrease. 

Our  analysis  is  concerned  with  how  pattern  search  behaves  in  the  neighborhood  of  an  isolated  local 
minimizer  x * .  We  make  the  following  assumptions  about  the  behavior  of  /  in  the  neighborhood  of  x* . 

Hypothesis  2.  We  assume  that  f  is  twice  continuously  differentiable  on  an  open  ball  B(x*,r})  of  x*, 
V/(x*)  =  0,  and  that  the  second  order  sufficiency  condition  V2/(x*)  >  0  holds  at  x*. 

We  further  assume  that  V2/(x)  is  positive  definite  for  all  x  €  £(x*,  rf).  Let  crm in  and  crmax  lower  and 
upper  bounds,  respectively,  on  the  singular  values  of  V2/(x)  on  B(x*,rj),  and  define  k  =  crmax/<7min. 

For  convenience,  we  will  assume  that  77  <  83.1,  where  £3.1  is  as  in  Theorem  8.1.  Clearly  we  may  do 
this  without  any  loss  of  generality.  This  assumption  ensures  that  B(x*,rj)  C  Q.  and  that  we  may  apply 
Theorem  3.1. 

Our  first  result  relates  A^  to  ||  Xk  —  x*  ||  at  unsuccessful  iterates. 

Proposition  4.1.  Under  Hypotheses  0-2,  there  exist  r]  >  0,  and  C4.1  >  0  for  which  the  following  holds. 
If  Xk  is  an  iterate  at  which  there  is  an  unsuccessful  iteration ,  A k  <  0,  ond  ||  Xk  —  ||  <  77,  then 

||  xk  -x*  ||  <  C4.1A*. 


Proof.  By  the  mean  value  theorem, 

Vf(xk)  -  V/(*.)  =  V2/(0(xfc  -  *.) 

for  some  £  on  the  line  segment  connecting  Xk  and  x*.  Since  V/(x*)  =  0,  we  have 
l|V/(xfc)||  =  ||V2/(£)(xfc  —  x*)||  >  amin||*fc-®*l|. 

Furthermore,  Theorem  3.1  holds  since  A^  <  77  <  <^3.1,  whence 

ffminll  xk  -  ||  <  II  V/(xfc)  ||  <  c3.iAfc. 

Setting  c4.i  =  C3.1/ <rmin  completes  the  proof.  □ 
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We  wish  to  prove  that  the  entire  sequence  of  iterates  converges  to  x*.  To  this  end,  we  begin  with  an 
elementary  result  concerning  the  level  sets  of  /  near  x * . 

Proposition  4.2.  Under  Hypothesis  2,  if  x,y  €  B(x*y  rf)  and  f(x)  <  f(y),  then 

(4.1)  II  x-x»  II  <  K^ll  y-x„  II, 

where  «  is  as  defined  in  Hypothesis  2. 

Proof.  Suppose  x,y  G  B{x*,rj)  and  /(x)  <  f{y).  Prom  Taylor’s  theorem  with  remainder  and  the  fact 
that  V/(x„)  =  0,  we  have 

f{y)  =  \(y  -  x*)Tv2f(0(y  -  x*) 

fix)  =  ^{x  -  x»)TV2/(w)(x  -  x») 

for  £  and  oj  on  the  line  segments  connecting  x*  with  y  and  x,  respectively.  Since  /(x)  <  f(y ),  we  obtain 
o  <  f(y)  -  f(x)  =  i(j /  -  x*)TV2f(0(y  -  x„)  -  ^{x  -  x,)TV2/(u;)(x  -  x»), 

whence 

0  ^  ^max||  y  ~~  X*  j|  <7min||  X  —  X*  |j  , 


and  thus  (4.1).  □ 

We  use  the  previous  proposition  to  show  that  if  we  start  sufficiently  close  to  x*  with  a  sufficiently  small 
pattern  size  parameter  A&,  and  we  stop  allowing  increases  in  A  &  after  some  point  (Hypothesis  1),  then 
pattern  search  will  not  move  away  from  a  neighborhood  of  x* . 

Proposition  4.3.  Under  Hypotheses  0-2,  there  exist  £4.3  >  0;  £4.3  >  0,  and  C4.3  >  0  for  which  the 
following  holds.  For  k>  N,  where  N  is  as  defined  in  Hypothesis  1 ,  if  Xk  is  an  iterate  for  which  A*  <  £4.3 
and  ||  Xk  —  x*  ||  <  £4.3,  then  for  all  i>k, 

||  XI  -  X*  ||  <  7] 


where  rj  is  as  defined  in  Hypothesis  2. 

Proof  Choose  64 .3  and  £4.3  to  satisfy 

<54,  <  f 

2co 

.  f »?  1  _i  ) 

£4.3  <  mm  -,-K  2  T)  ■ 

The  proof  is  by  induction.  First  consider  Xfc+i  =  Xk  +  sk-  By  Hypothesis  0,  there  is  a  constant  Co  such 
that  ||  Xfc+i  -  xk  ||  =  ||  sk  ||  <  c0Ajfc.  We  have,  a  priori , 

II  Xk+ 1  -  X*  II  <  II  Xfc+i  -  Xfc  ||  +  ||  Xfc  -  x»  ||  <  CoAfc  +£4.3  <  V, 

so  Xfc+1  €  B(xt,rj).  Since  /(x*+i)  <  f(xk),  and  xfe+i  €  B(x*,»?),  from  Proposition  4.2  we  obtain 

II  Xfc+i  X*  |j  <  K?  ||  Xk  X*  ||  <  K*£ 4.3  <  fj. 

Now  consider  any  £  >  k  +  1,  and  suppose 

||  XI  -  x,  II  <  Tj. 
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Then 


(4.2)  ||  xz+i  -  x*  ||  <  ||  xi+x  -  xe  ||  +  ||  X£  -  x*  || . 

Hypothesis  1,  assures  us  that  A^  <  A*,  for  £  >  k,  so 

||  xe+i  -  xt  ||  <  coA^  <  coAfc. 

Meanwhile,  by  the  induction  hypothesis,  x^  €  B(x*,r}).  Since  f(xt)  <  f(xk),  as  well,  Proposition  4.2  and 
the  assumption  ||  Xk  -  x*  ||  <  64.3  say  that 

II  Xt  —  X*  ||  <  ||  Xk  —  X*  ||  <  3. 


Thus  (4.2)  yields 


||  xt+i  -  x*  ||  <  coAfc  +  «2e4.3  <  rj . 


0 

For  the  final  local  convergence  result,  we  need  the  following  additional  hypothesis,  which  underlies  the 
global  convergence  analysis  of  pattern  search  [11]. 

Hypothesis  3.  The  set  L(x 0)  is  compact 

From  [11]  we  know  that  if  L(x 0)  is  compact,  lim  inffc-^oo  Ak  =  0.  Thus,  Hypothesis  1  and  Hypothesis  3 
together  mean  that  limfc—oo  A&  =  0. 

Putting  the  pieces  together,  we  obtain  the  following  local  convergence  result  for  pattern  search. 
Theorem  4.4.  Suppose  Hypothesis  3  holds.  Given  a  pattern  search  algorithm  satisfying  Hypotheses 
0-1 ,  suppose  there  exists  a  limit  point  x*  of  the  sequence  of  iterates  {xk}  produced  by  the  algorithm  that  is 
a  local  minimizer  satisfying  Hypothesis  2 . 

Then  there  exist  c4.4  >  0  and  K  such  that  for  all  k>  K, 

(4.3)  ||  Xk  ~  x*  ||  ^  c4.4Am(fc), 

where  m(k)  is  the  last  unsuccessful  iterate  preceding  k.  As  a  consequence ,  we  have  lim^oo  Xk  =  x*. 

Proof  Recall  N  from  Hypothesis  1,  the  iteration  after  which  we  no  longer  allow  Afc  to  increase.  Since 
x*  is  a  limit  point  of  {xk}  and  limfc_>oo  Ak  =  0  (by  Hypotheses  1  and  3),  there  exists  an  iterate  x^,  ki  >  N 
such  that 


Akl  <  min{7?,64.3} 

II  xkl  -x*  ||  <  min{?7,  £4.3}, 

where  r)  is  as  in  Hypothesis  2  and  <S4.3,  £4.3  are  as  in  Proposition  4.3. 
By  Proposition  4.3,  we  have 


||  Xk  ~  X*  ||  <  7] 

for  all  k>  ki>  Then,  by  Proposition  4.1,  we  have 


II  Xffi  x*  ||  ^  c4,iA  m 


for  all  unsuccessful  iterates  xm  with  m>k\. 
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Now  let  K  be  the  first  iteration  after  k\  at  which  we  have  an  unsuccessful  iteration.  For  all  unsuccessful 
iterations  k>  K,  Proposition  4.1  gives  us 

(4.4)  ||  Xfc  X *  ||  ^  C4.1^fc 

since  k  —  m(k)  in  this  case.  Meanwhile,  for  the  successful  iterations  k  >  K,  we  have  f(xk)  <  /(^m(fc))?  an^ 
Proposition  4.3  assures  us  that  ||  Xk  —  x*  ||  <  rj.  It  then  follows  from  Proposition  4.2  that 

(4.5)  ||  x*  -  x*  ||  <  ||  xmQ b)  -  x*  ||  <  «ic4.iAm(A.). 

Then  (4.3)  follows  from  (4.4)  and  (4.5),  with  C4.4  =  K^c^i- 

Since  Hypotheses  1  and  3  imply  A k  — ►  0,  it  follows  that  lim^  — >oo  %k  —  X* .  D 

Theorem  4.4  complements  Theorem  3.7  of  [11],  where  it  is  shown,  under  different  hypotheses  and  a  more 
stringent  criterion  for  accepting  a  step,  that  ||  V/(xfc)  ||  — >  0. 

Theorem  4.4  says  that  for  the  subsequence  of  unsuccessful  iterates,  the  rate  of  convergence  is  r-linear. 
Theorem  4.4  says  nothing  about  what  may  happen  at  the  successful  iterations,  nor  how  many  such  itera¬ 
tions  there  may  be  between  unsuccessful  iterations.  What  we  have  is  a  sort  of  multi-step  r-linear  rate  of 
convergence,  but  one  for  which  we  do  not  know,  and,  as  our  numerical  tests  indicate,  cannot  predict,  what 
the  number  of  intervening  steps  may  be.  For  want  of  an  existing  term  for  this  notion  of  convergence,  we  call 
it  desultory  r-linear  convergence . 

The  obstruction  to  sharpening  the  rate  of  convergence  result  is  that  we  do  not  know  a  priori  how  much 
improvement  we  obtain  in  f(x)  at  the  successful  iterations.  At  unsuccessful  iterates,  we  reach  the  decision 
that  we  can  only  make  progress  by  reducing  A*  only  after  we  evaluate  /(x)  along  a  set  of  directions  that 
necessarily  includes  a  descent  direction  (see  Fig.  2.3  and  Proposition  2.1).  On  the  other  hand,  successful 
iterates  are  less  informative.  We  have  no  a  priori  idea  of  how  much  improvement  we  obtain  in  a  successful 
iteration  relative  to  the  minimization  of  the  quadratic  model  of  f(x)  along  the  pattern  search  directions. 
Given  the  paucity  of  information  generally  available  in  pattern  search  one  must  not  expect  too  much. 

More  positively,  Theorem  4.4  suggests  how  one  can  “accelerate”  the  local  convergence  of  pattern  search 
algorithms.  One  need  only  rename  the  formerly  unsuccessful  iterates  successful  iterates  and  drop  the  formerly 
successful  iterates  from  discussion.  Then,  mirabile  dictu ,  this  simple  modification  makes  the  successful 
iterations  an  r-linear ly  convergent  sequence. 

5.  Numerical  results.  We  now  present  some  numerical  experiments  that  illustrate  the  practical  im¬ 
plications  of  the  local  convergence  results.  The  first  round  of  testing,  reported  in  Section  5.2,  supports  the 
analysis;  the  second  round,  reported  in  Section  5.2,  shows  its  limitations. 

The  numerical  results  we  report  are  in  no  way  exhaustive,  and  simply  serve  to  illustrate  how  the  local 
convergence  results  are  manifest  in  practice.  The  results  we  report  here  on  the  effectiveness  of  A&  as  a 
measure  of  stationarity  are,  in  part,  a  summary  of  some  of  the  results  reported  in  [5].  The  second  round  of 
testing  regarding  the  local  rate  of  convergence  was  based  on  the  implementation  of  pattern  search  developed 
for,  and  reported  in,  [5]. 

5.1.  The  testing  environment.  Full  details  of  the  numerical  experiments  can  be  found  in  [5].  The 
tests  we  report  here  were  done  with  randomly  generated  positive  definite  quadratic  functions.  This  is  a 
reasonable  choice,  since  we  are  interested  in  the  local  convergence  behavior  of  pattern  search,  and  any  C 2 
function  looks  like  a  convex  quadratic  in  the  neighborhood  of  an  isolated  local  minimizer.  The  quadratics 
tested  were  of  the  form  /(x)  =  xT Ax  +  c,  where  A  =  HTH  and  H  €  R(n+2)xn  is  a  matrix  with  entries 
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that  are  normal  random  variates  with  means  of  zero  and  standard  deviations  of  one.  The  lack  of  a  linear 
term  causes  the  solution  to  lie  at  the  origin.  The  constant  term  c  is  not  interesting  for  the  purposes  of  the 
optimization,  but  provides  a  useful  tag  for  identifying  individual  functions.  For  the  testing  in  [5],  2  <  n  <  5; 
we  show  two  results  for  n  =  5. 

In  addition  to  randomly  generating  the  entries  of  the  matrix  H,  we  also  randomly  generated  Ao  and  the 
entries  of  the  vector  xq.  The  entries  for  the  starting  point  xq  were  also  normal  random  variates  with  means 
of  zero  and  standard  deviations  of  one.  The  choice  for  Ao  was  an  exponential  variate  with  a  mean  of  one. 

The  software  described  in  [5]  was  written  in  C++  to  make  use  of  C++  classes,  a  convenient  way  to 
establish  the  key  features  of  pattern  search  and  then  easily  derive  specific  variants.  Several  of  these  variants 
were  implemented  and  tested,  as  described  in  [5].  We  show  results  using  HJSearch,  an  implementation  of 
the  classical  pattern  search  algorithm  of  Hooke  and  Jeeves  [6];  CompassSearch,  the  pattern  search  algorithm 
described  in  Section  2;  and  NLessSearch,  a  pattern  search  algorithm  that  takes  advantage  of  the  fact  that  a 
minimal  positive  basis  requires  only  n  +  1  vectors  [7],  as  opposed  to  the  2 n  coordinate  vectors  used  in  most 
traditional  pattern  search  methods,  including  compass  search  and  Hooke  and  Jeeves. 

5.2.  Measuring  stationarity.  The  first  question  we  ask  is  how  effective  is  A&  as  a  measure  of  station- 
arity?  Not  too  surprisingly,  the  results  of  our  tests  showed  that  A^  is  a  reliable  measure  of  progress  toward 
a  solution.  Furthermore,  our  numbers  make  quite  clear  the  r-linear  behavior  of  the  sequence  of  unsuccessful 
iterates. 

After  any  unsuccessful  iteration,  a  pattern  search  method  is  required  to  reduce  A We  used  the  standard 
reduction  factor  of  \  so  that  after  an  unsuccessful  iteration,  Ak+i  =  ^A Before  proceeding  to  the  next 
iteration,  we  recorded  the  value  of  A/e,  ||V/(xfc)||,  \f(xk)  —  f(x*) |>  and  U#*  —  z*||  (though  since  we  knew 
x *  =  0,  we  simply  had  to  compute  \\xk\\)>  Representative  results  from  two  particular  tests  are  given  in 
Tables  5.1  and  5.2. 


Table  5.1 

NLessSearch  in  five  variables 


A  k 

0.696226813823902 

0.348113406911951 

0.174056703455976 

0.087028351727988 

0.043514175863994 

0.021757087931997 

0.010878543965999 

0.005439271982999 

0.002719635991500 

0.001359817995750 

0.000679908997875 

0.000339954498938 

0.000169977249469 

0.000084988624734 

0.000042494312367 

0.000021247156184 

0.000010623578092 

0.000005311789046 


l|v/(sfc)ll 

3.868198250940641 

1.774568869170108 

1.453229516907024 

0.281730564593278 

0.294368401951189 

0.139178781830330 

0.029890455941604 

0.029890455941604 

0.026442509744175 

0.012307380822946 

0.009511207300009 

0.005426023491341 

0.002065061853736 

0.001038174548727 

0.000435621809509 

0.000231605471572 

0.000113220891634 

0.000067613953149 


0.97865190917639 

0.29434050891286 

0.14479398150378 

0.02631186626609 

0.01457817127405 

0.00462725481091 

0.00073483024738 

0.00073483024738 

0.00033346616931 

0.00022911316873 

0.00014689251267 

0.00004796415094 

0.00000720339894 

0.00000179297966 

0.00000034070778 

0.00000008825774 

0.00000001411259 

0.00000000577676 


_ \\xk  x*\\ _ 

0.277727609425140 

0.217654912583381 

0.108469794156683 

0.115903974197448 

0.070417918465458 

0.040571903721886 

0.034760919692137 

0.034760919692137 

0.024811502814971 

0.021849036278915 

0.017596624227294 

0.010013823809977 

0.003890485486089 

0.001938237809581 

0.000849356148369 

0.000432171965702 

0.000168890905210 

0.000108943399928 
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Table  5.2 

HJSearch.  in  five  variables 

A fc  ||V/(jgfc)H _ \f{xk)-f(x*)\ _ IN  ~x*\\ 


0.696226813823902 

0.348113406911951 

0.174056703455976 

0.087028351727988 

0.043514175863994 

0.021757087931997 

0.010878543965999 

0.005439271982999 

0.002719635991500 

0.001359817995750 

0.000679908997875 

0.000339954498938 

0.000169977249469 

0.000084988624734 

0.000042494312367 

0.000021247156184 

0.000010623578092 

0.000005311789046 


3.718628968450993 

1.370661155865317 

0.993386770046628 

0.236893510661273 

0.314026005456998 

0.131650296045321 

0.042526372212693 

0.032921235371376 

0.012854930063180 

0.005667414556147 

0.004101406209192 

0.001396318029208 

0.000833651146770 

0.000563050121378 

0.000112117511534 

0.000097664692564 

0.000035578092711 

0.000010624256256 


3.96639084353257 

0.44618879006458 

0.19091214014793 

0.01477525409286 

0.01421309666224 

0.00223337373949 

0.00028996796577 

0.00018078437086 

0.00014567060113 

0.00001023084696 

0.00000429756349 

0.00000050775161 

0.00000049903818 

0.00000002356890 

0.00000000325088 

0.00000000266601 

0.00000000026108 

0.00000000015362 


2.396301558944381 

0.698389592313846 

0.450386903073632 

0.153943970082610 

0.119315177505950 

0.034002609804365 

0.015791910616849 

0.014678346820778 

0.016582990396810 

0.003757596625046 

0.002612852810391 

0.000854609846084 

0.000985750547712 

0.000074244150774 

0.000043510325021 

0.000032689236837 

0.000013878637584 

0.000017458315183 


The  purpose  of  the  results  we  report  in  Tables  5.1  and  5.2  is  not  to  demand  close  scrutiny  of  each  entry, 
but  rather  to  demonstrate  the  trends  in  each  of  the  four  quantities  measured.  We  clearly  see  the  r-linear 
behavior  the  analysis  tells  us  to  expect:  by  the  time  we  halve  Afc,  we  have  roughly  halved  the  error  in  the 
solution.  (Note  that  in  Table  5.1  we  have  two  consecutive  unsuccessful  iterations.) 

We  report  here  the  results  from  only  two  experiments,  but  these  are  representative  of  results  from  ten 
thousand  runs  over  multiple  quadratics,  in  multiple  dimensions,  from  multiple  starting  points,  with  multiple 
choices  of  Ao,  using  four  different  pattern  search  methods.  We  found  that  across  all  these  tests  A*  gave  us 
a  consistent  measure  of  the  accuracy  of  the  solution. 

One  beauty  of  using  Ak  as  a  measure  of  stationarity  is  that  it  is  perfectly  natural  in  the  context  of 
pattern  search;  no  additional  computation  is  required.  Another  strong  reason  for  using  Ak  as  a  measure 
of  stationarity  is  that  it  is  remarkably  insusceptible  to  error.  Once  an  initial  choice  of  Ak  has  been  made, 
the  only  possible  numerical  error  occurs  when  converting  the  decimal  representation  to  binary.  Since  most 
implementations  use  the  value  of  \  to  reduce  A&,  which  requires  only  a  simple  binary  shift,  there  are  no 
additional  numerical  issues  to  cloud  the  computation  of  this  measure.  We  close  with  one  final  observation  to 
be  made  about  the  practical  utility  of  A^  as  a  measure  of  stationarity.  A  useful  feature  of  pattern  search  is 
that  one  only  requires  ranking,  or  order,  information  to  drive  the  search — no  numeric  values  for  the  function 
are  necessary  [7].  In  such  a  setting,  Ak  is  a  feasible  measure  of  progress  whereas  measures  based  on  the 
numeric  values  of  the  objective  function  are  not. 

6.  How  many  successful  iterates?  Theorem  4.4  says  that  the  subsequence  of  unsuccessful  iterates 
converges  r-linearly  once  we  are  in  the  neighborhood  of  a  solution.  A  natural  question  to  then  ask  is:  how 
many  iterations  occur  in  practice  between  each  iterate  included  in  this  subsequence?  This  question  is  more 
subtle,  and  our  results  are  less  conclusive.  Again,  we  illustrate  our  findings  with  only  a  few  specific  examples 
in  Figs.  6.2-6.5. 
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In  all  instances,  we  give  a  final  stopping  tolerance  of  2  x  10“8;  i.e.,  we  terminate  the  search  when 
Afc  <  2  x  10~8.  Along  the  horizontal  axis,  we  list  the  number  of  unsuccessful  iterations;  i.e.,  the  number 
of  times  we  halve  Ak  before  it  is  less  than  the  stopping  tolerance.  Each  bar  then  represents  the  number  of 
successful  iterations  that  preceded  an  unsuccessful  iteration  plus  the  (single)  unsuccessful  iteration  so  that 
summing  all  the  entries  gives  us  the  total  number  of  iterations  for  the  search. 

Notice  that  for  the  three  algorithms  we  tested,  the  scale  on  the  vertical  axes  varies  considerably.  For 
NLessSearch,  the  number  of  successful  iterations  preceding  an  unsuccessful  iteration  can  be  considerably 
higher  than,  say,  for  HJSearch,  but  overall,  the  results  are  inconclusive.  We  cannot  predict  how  many 
successful  iterations  may  proceed  an  unsuccessful  iteration,  nor  does  there  seem  to  be  any  particular  trend. 

The  only  substantive  observation  we  can  extract  from  our  tests  is  illustrated  by  the  example  shown  in 
Fig.  6.1. 


Hjrribt*  of  Local  Seart*  Iterations  for  oech  Decrsaae  in  Delta  Number  of  Local  Seerdi  Iterations  for  oadi  Decrease  in  Dele  Number  of  Local  Search  Iterations  for  each  Decrease  In  Data 


Fig.  6.1.  NLess,  Compass,  &  HJ  in  4  variables 


Here  the  explanation  for  the  huge  number  of  successful  iterations  before  Ak  is  ever  reduced  seems  to  be 
straightforward.  The  initial  choice  of  Ao,  which  is  drawn  randomly,  is  so  small  (0.001128116614106)  that 
initially  there  is  a  long  sequence  of  successful  iterations,  but  progress  is  remarkably  slow  because  we  start 
with  such  a  small  choice  of  Ao  that  all  the  trial  steps  are  quite  short.  The  obvious  conjecture  to  make  is 
that  it  is  better  to  start  with  a  reasonably  large  choice  of  Ao — a  practical  issue  that  is  the  subject  of  another 
study  in  preparation. 


Numb*  of  Loot  Search  Iteration*  for  each  Decrease  In  Delta  Number  of  Local  Search  Iteration*  (or  each  Decrease  In  Deta  Number  cf  Local  Search  Iterators  for  each  Decrease  in  Do*e 


Fig.  6.2.  NLess,  Compass,  &  HJ  in  8  variables 


Fig.  6.3.  NLess,  Compass,  &  HJ  in  8  variables 
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Nunbsr  of  locsl  Swdi  Iterations  for  wcJi  Decrees*  in  D«t* 


Number  of  Local  Search  Iteration*  for  cedi  Decrease  m  Data 


Number  of  Local  Search  Iterations  for  each  Decrease  In  Deta 


Fig.  6.4.  NLess,  Compass ,  &  HJ  in  4  variables 


Number  of  Local  Search  Iterations  for  each  Decrease  in  Defta 


Number  of  Local  Search  lleratlona  tor  each  Decreaae  In  Delta 


Number  of  Local  Search  Iterations  for  each  Decrease  in  Deta 


Fig.  6.5.  NLess,  Compass,  &  HJ  in  4  variables 


7.  Conclusion.  The  results  given  here  round  out  the  convergence  analysis  of  pattern  search.  The 
analysis  and  numerical  experiments  reported  here  show  that  A k  can  be  used  as  a  reliable  stopping  criterion. 
Moreover,  these  tests  show  that  the  correlations  predicted  by  Theorems  3.1  and  4.4  between  Afc,  H  V/(xfc)  ||, 
and  ||  Xk  —  x*  ||  are  manifest  in  practice.  These  results  vindicate  the  intuition  of  the  early  developers  of 
direct  search  methods. 
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